As background, in the remote sensing/aerial imaging industry, imagery may be used to capture views of a geographic area. The imagery may be used to measure objects and structures within images, as well as, to be able to determine geographic locations of points within images.
Geographic information about objects within an image may be associated with that image. Such images are generally referred to as “geo-referenced images”. Geo-referenced images may include two basic categories: captured imagery (images as captured by the camera or sensor employed), and projected imagery (images processed and converted to confirm to a mathematical projection).
Geo-referenced aerial images may be produced using hardware and/or software systems that may geo-reference airborne sensor data. For example, methods and apparatus for mapping and measuring land are described in U.S. Pat. No. 5,247,356, which is hereby incorporated by reference in its entirety. In addition, a system produced by Applanix Corporation of Richmond Hill, Ontario, Canada and sold under the trademark “POS AV” includes a hardware and software system for directly geo-referencing sensor data. This system may be mounted on a moving platform, such as an airplane, and directed towards the ground.
Imagery may begin as captured imagery. The captured imagery may need further processing to create projected imagery that is geo-referenced. The conventional method for processing captured imagery into projected imagery is ortho-rectification. Ortho-rectification aligns an image to an orthogonal or rectilinear grid (i.e., composed of rectangles). Captured imagery used to create an ortho-rectified image may typically include a nadir image—that is, an image captured with the camera pointing straight down.
Direct geo-referencing is the direct measurement of sensor position and orientation (e.g., exterior orientation parameters), without the need for additional ground information over the project area. These parameters may include data from an airborne sensor that may be geo-referenced to the Earth and/or local mapping frame. Examples of airborne sensors may include: aerial cameras (digital or film-based), multi-spectral or hyper-spectral scanners, SAR, or LIDAR.
Geographical location data and/or geospatial data may be stored, organized, and/or analyzed in a Geographical Information System (hereinafter “GIS” or “GISs”). In aerial mapping, captured aerial images may be warped to fit a pre-defined mapping grid (e.g., U.S. State Plane, 1983 North American Datum, in U.S. Survey Feet). When an image frame is displayed, geographical bounds of that image frame may be used to retrieve GIS data in that area. Each geographic point location may be then translated from geographic coordinates (e.g., latitude/longitude, X/Y coordinates) to image frame coordinates (e.g., pixel row/column) using mapping information surrounding the image frame. For traditional nadir imagery, translation from geographic coordinates to image coordinates may be fairly straight forward as the image may be warped to fit a mapping grid (e.g., using an ortho-rectification process). For oblique imagery, however, such translation may be more complex, and computation-intensive as some three dimensional features may become distorted during image processing.
Multiple captured images may also be combined into one or more larger composite images. The larger composite images may cover larger geographic areas. For example, the composite image may be an ortho-mosaic image created from a series of overlapping or adjacent captured nadir images. The overlapping or adjacent images may be mathematically combined into a single ortho-rectified processed composite image.
Generally, in creating an ortho-mosaic image a rectilinear grid may be created. For example, the rectilinear grid may include an ortho-mosaic image, wherein every grid pixel covers the same amount of area on the ground. The location of each grid pixel may be determined from the mathematical definition of the grid. Generally, this means the grid may include a starting or origin location (e.g., X and Y location), and a grid/pixel size (e.g., X and Y grid/pixel size). As such, the location of any pixel may be determined by:XOriginXSize×XColumn Pixel=XPixel  (EQ. 1)YOriginYSize×XRow Pixel=YPixel  (EQ. 2)The available nadir images may be evaluated to determine if the images cover the same point on the ground as the grid pixel being filled. If so, a mathematical formula may be used to determine where that point on the ground projects up onto the camera's pixel image map, and that resulting pixel value may be then transferred to the grid pixel.
While the above methodology may be applied to individual video frames, the ability to geo-reference and overlay GIS data in real-time at full motion video frame rates has not been achieved by currently available systems for several reasons. For example, the ortho-rectification procedure may be highly computation-intensive (e.g., elevation data). Even further, the computational demands increase exponentially as the frame rate increases. For the frame rate required for full motion video (e.g., approximately twelve to thirty frames per second), the computational requirements make a real-time system impractical.
Current art, due to its computational limitations, may store a single geographic position for each frame of a video. As such, the video may be found in a GIS data search, however, there may be limitations for geographical location determinations for each pixel in the video frame(s). Additionally, such limits may not include measurement of distances between objects in the video frame(s) and/or overlay of GIS data over a series of video frame(s) at full motion video rates in real-time.
Existing systems overlaying information onto full motion video streams in real-time may operate by calibrating to specific targets. For example, a fan of the National Football League may be familiar with overlay graphics on the line of scrimmage, the first down marker, and the like. Such systems work, not through geo-referencing of the imagery, but by calibrating the cameras to the field in that specific football stadium, and including manual information input for the computers to then overlay on the video stream via chroma-key methodology. If the cameras are pointed anywhere but that particular football field for which they are calibrated, the overlays may not be at the correct location because the images are not georeferenced.
A recent image processing technique, introduced by Pictometry International Corp., warps a grid to an image instead of warping the image to fit the grid. This is especially interesting for oblique image processing, as oblique images (i.e., non-nadir images) may typically introduce gross three dimensional object distortions when warped to fit a mapping grid. Further, the development of a tessellated ground plane includes a means to define the surface of the Earth under an oblique image. The systems and methods for determining tessellated ground planes are further described in detail in U.S. Pat. No. 7,424,133, which is hereby incorporated by reference in its entirety. By capturing all of the interior and exterior parameters surrounding the image, Pictometry may be able to determine locations, derive measurements, and/or overlay GIS data all with a degree of accuracy previously unachieved for oblique imagery.
Another recent approach by Pictometry International Corporation includes the systems and methods for single ray projection also described in U.S. Pat. No. 7,424,133. These methods, while more accurate than ortho-rectification, may be too slow for real-time processing at full motion video frame rates.